__author__ = 'Stephanie Juneau <stephanie.juneau@noirlab.edu>, David Herrera <david.herrera@noirlab.edu>, and the Astro Data Lab Team <datalab@noirlab.edu>'
__version__ = '20230615' # yyyymmdd
__datasets__ = ['ls_dr3','sdss_dr13']
__keywords__ = ['extragalactic','galaxies','joint query','spectroscopic redshift','3d plot']

DECaLS and SDSS/BOSS Large Scale Structure

by Stéphanie Juneau, David Herrera, and the Astro Data Lab Team

Table of contents

• Goals & Summary
• Disclaimer & attribution
• Imports & setup
• Joint Query of LS and SDSS catalogs
• Plot Results
• Exercise
• Large scale structure in 3D

Goals

• Joint query between photometric (LS) and spectroscopic (SDSS) catalogs
• Plot on-sky position of extragalactic objects, color-coded by redshift
• Plot positions of galaxies in 3D (real space), and explore large scale structure interactively

Summary

In this Notebook, we explore large-scale structures of galaxies by combining spectroscopic redshifts from SDSS/BOSS with photometry from the DESI pre-imaging Legacy Survey (LS). The advantage of spectroscopic redshifts is that they are far more accurate than photometric redshifts to probe distances to galaxies (though still need to be corrected for possible distortion effects such as the finger-of-God effect, which we ignore here). The advantage of the LS photometry is that it reaches deeper than SDSS by about 1 magnitude, which yields better image quality to measure magnitudes, colors, and galaxy shapes. While there are several possible extensions to the example work included below, we will show that a simple figure of galaxy spatial locations color-coded by galaxy morphological type reveals the known morphology-density relation.

We wanted to extend indeed a little further and be able to visualize and even interact with a representation of these galaxies in the actual space. For that, we developed a 3D plot based directly (as a flat cube, without any projection or correction) on RA, DEC and z, and that is at the end of this NB.

On a technical point of view, this short notebook illustrates an example joint query between the LS DR3 photometry Tractor table, and the SDSS/BOSS DR13 specObj spectroscopy table. It takes advantage of the fact that there is a version of the LS DR3 tractor table that was pre-matched to SDSS/BOSS DR13 so we can join on a common column called specobjid.

The columns from the LS table used (Tractor, pre-matched to specObj DR13) can be seen here: http://datalab.noirlab.edu/query.php?name=ls_dr3.dr3_specobj_dr13

The columns from the SDSS/BOSS DR13 table can be seen here: http://datalab.noirlab.edu/query.php?name=sdss_dr13.specobj

The columns from the SDSS DR17 used for the 3D plot can be found here http://datalab.noirlab.edu/query.php?name=sdss_dr17.specobj

Disclaimer & attribution

If you use this notebook for your published science, please acknowledge the following:

• Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, http://dx.doi.org/10.1117/12.2057445

• Data Lab disclaimer: http://datalab.noirlab.edu/disclaimers.php

Imports and setup

Please note that this notebook is written for Python 3.

# std lib
from getpass import getpass

# 3rd party
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.patches as patches
from scipy.stats import binned_statistic_2d
%matplotlib inline
from astropy.table import Table
from astropy.cosmology import Planck18 as cosmo
import plotly
import plotly.graph_objs as go
import pandas as pd
plotly.offline.init_notebook_mode()

# Data Lab
from dl import queryClient as qc
from dl import authClient as ac

Authentication

Much of the functionality of Data Lab can be accessed without explicitly logging in (the service then uses an anonymous login). But some capacities, for instance saving the results of your queries to your virtual storage space, require a login (i.e. you will need a registered user account).

If you need to log in to Data Lab, un-comment the cell below and execute it:

#token = ac.login(input("Enter user name: (+ENTER) "),getpass("Enter password: (+ENTER) "))
#ac.whoAmI()

Query DECaLS Tractor Photometry Catalog

The photometry is derived from Tractor modeling of sources, and the database includes model photometry, type (shape), as well as other quantities.

The Legacy Survey DR3 database is called ls_dr3 and includes several tables. We will use the dr3_specobj_dr13 table, which is a version of the LS DR3 tractor table which was pre-matched with specObj table from SDSS/BOSS DR13. The column names and descriptions can be found from the Data Lab Query Interface or using the Table Access Protocol (TAP) service with, e.g., TOPCAT. In both cases, we are interested in ls_dr3.dr3_specobj_dr13.

The SDSS DR13 database is called sdss_dr13 and also includes several tables. We will use the specObj table, which has spectroscopic information.

%%time
# number of rows from LS DR3 tractor (NOTE: tractor is the main photometry table):
query="SELECT nrows FROM tbl_stat WHERE schema='ls_dr4' and tbl_name='tractor'"

# Call query manager
response = qc.query(sql=query, fmt='csv')

print(response)

nrows
187967232

CPU times: user 17.8 ms, sys: 1.02 ms, total: 18.9 ms
Wall time: 96.6 ms

%%time
# number of rows from SDSS specObj DR13:
query="SELECT nrows FROM tbl_stat WHERE schema='sdss_dr13' and tbl_name='specobj'"

# Call query manager
response = qc.query(sql=query, fmt='csv')

print(response)

nrows
4411200

CPU times: user 22.2 ms, sys: 3.38 ms, total: 25.6 ms
Wall time: 87.2 ms

# ls_dr3.dr3_specobj_dr13           # DECaLS matched to SDSS DR13 specobj
# sdss_dr13.specobj                 # SDSS DR13 specobj

# Write query statement (adql)
query = ("""
  SELECT L.ra, L.dec, L.type, L.g_r, L.r_z,
         S.z, S.plug_ra, S.plug_dec, S.class, S.vdisp, S.vdisp_err 
  FROM ls_dr3.dr3_specobj_dr13 AS L
  JOIN sdss_dr13.specobj AS S ON L.specObjId = S.specobjid 
  WHERE L.ra BETWEEN %s AND %s AND L.dec BETWEEN %s AND %s AND (L.ra_ivar > 0) 
  LIMIT 100000
  """) % (126,131,7.,12.)  # small region

# L.ra, L.dec        = RA, Dec from Legacy Survey (LS) table    
# L.type             = object type (PSF, SIMP, EXP, DEV, COMP)
# L.g_r, L.r_z       = pre-computed g-r and r-z colors from photometry
# S.z                = redshift (z) from SDSS specObj table
# S.plug_ra, dec     = RA, Dec of SDSS fiber from specObj table
# S.class            = Source class (Star, Galaxy, QSO) from SDSS
# S.vdisp, vdisp_err = velocity dispersion (and error) from SDSS specObj table
#
# WHERE: requirement that RA & Dec coordinates are within a rectangular region

print(query)

  SELECT L.ra, L.dec, L.type, L.g_r, L.r_z,
         S.z, S.plug_ra, S.plug_dec, S.class, S.vdisp, S.vdisp_err 
  FROM ls_dr3.dr3_specobj_dr13 AS L
  JOIN sdss_dr13.specobj AS S ON L.specObjId = S.specobjid 
  WHERE L.ra BETWEEN 126 AND 131 AND L.dec BETWEEN 7.0 AND 12.0 AND (L.ra_ivar > 0) 
  LIMIT 100000
  

%%time
# Call query manager
response = qc.query(adql=query, fmt='csv')

CPU times: user 38.6 ms, sys: 4.16 ms, total: 42.8 ms
Wall time: 861 ms

# Reformat output into a table
result = Table.read(response, format='csv')
result[:10]

Table length=10

ra	dec	type	g_r	r_z	z	plug_ra	plug_dec	class	vdisp	vdisp_err
float64	float64	str4	float64	float64	float64	float64	float64	str6	float64	float64
126.0000791669008	7.034954345152388	DEV	1.331186	0.810879	0.212844	126.00008000000003	7.0349361	GALAXY	284.4305	15.617004
126.0149385106863	7.007749700524829	DEV	1.415909	0.830248	0.243859	126.01495	7.0077364	GALAXY	229.53995	23.60804
126.02662496545437	7.04572519907787	DEV	1.477915	0.841503	0.259155	126.02663000000001	7.0457033	GALAXY	221.32855	12.791714
126.02401722142885	7.067081473462699	EXP	1.084211	0.941747	0.063915	126.02422	7.0671055	GALAXY	144.77763	10.631522
126.0135291447847	7.072400035246026	COMP	0.974909	0.730096	0.064388	126.01356	7.0723868	GALAXY	149.01518	12.994181
126.00355789210869	7.10391918975098	PSF	0.825657	0.744955	1.289764	126.00357	7.1038999	QSO	0.0	0.0
126.0837804508805	7.062649354011356	DEV	1.622526	0.875732	0.31347	126.08379	7.0626316	GALAXY	273.91946	22.274921
126.11028773792533	7.059580140782136	PSF	-0.309608	0.802147	2.227892	126.11029000000002	7.0595677	QSO	0.0	0.0
126.1333967060125	7.0422990856476755	PSF	0.279249	0.08976	0.00027	126.13341	7.0422755	STAR	0.0	0.0
126.13860749018284	7.043995358472932	DEV	1.963074	1.209225	0.509449	126.13862	7.0439339	GALAXY	349.83282	87.22228

Plot Results

Sanity check: RA, Dec positions from both tables

# convert RA coordinates from [0,360] to [-180,180] 
chgsign = np.where(result['ra'] > 180)
result['ra'][chgsign] = result['ra'][chgsign]-360.
result['plug_ra'][chgsign] = result['plug_ra'][chgsign]-360.

plt.figure(figsize=(9,8))

# plot RA, Dec from LS catalog in red with larger symbols
plt.scatter(result['ra'],result['dec'],s=3,color='red',marker='1')

# overplot RA, Dec from SDSS catalog in blue with smaller symbols
plt.scatter(result['plug_ra'],result['plug_dec'],s=3,color='black',marker='2')

# Extent of RA, Dec (in degrees) to plot
xmin = 126.
xmax = 131.
ymin = 7.
ymax = 12.

plt.axis([xmin, xmax, ymin, ymax])
plt.xlim(reversed(plt.xlim())) # flip the x-axis
plt.xlabel("RA (degrees)", fontsize=20)
plt.ylabel("Dec (degrees)", fontsize=20)
plt.show()

Visual Inspection of Large-Scale Structures

Plot the positions of a broad range of redshift, and overplot a thin slice in redshift to show possible structures within that slice.

# Select redshift slice
rz = np.where((result['z'] >0.105) & (result['z']<0.125))

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6.5))

# plot all points in red (all redshifts)
ax1.scatter(result['plug_ra'],result['plug_dec'],s=3,color='r',marker='o',alpha=0.25)

# overplot in blue objects in narrow redshift slice
ax1.scatter(result['plug_ra'][rz],result['plug_dec'][rz],s=10,color='black')

# Extent of RA, Dec (in degrees) to plot
xmin = 126.
xmax = 131.
ymin = 7.
ymax = 12.

ax1.axis([xmin, xmax, ymin, ymax])
ax1.set_xlim(reversed(ax1.set_xlim())) # flip the x-axis
ax1.set_xlabel("RA (degrees)", fontsize=20)
ax1.set_ylabel("Dec (degrees)", fontsize=20)

# add rectangle to show where we zoom in next panel
ax1.add_patch(patches.Rectangle((128.65-0.25, 8.85-0.2),0.5,0.4,fill=False,color='b'))


## ZOOM IN A SMALLER REGION

# plot all points in red (all redshifts)
ax2.scatter(result['plug_ra'],result['plug_dec'],s=15,color='r',marker='o',alpha=0.3)

# overplot in blue objects in narrow redshift slice
ax2.scatter(result['plug_ra'][rz],result['plug_dec'][rz],s=30,color='black')

# Extent of RA, Dec (in degrees) to plot
xmin = 128.4
xmax = 128.9
ymin = 8.65
ymax = 9.05

ax2.axis([xmin, xmax, ymin, ymax])
ax2.set_xlim(reversed(ax2.set_xlim())) # flip the x-axis
ax2.set_xlabel("RA (degrees)", fontsize=20)
ax2.set_ylabel("Dec (degrees)", fontsize=20)

# add rectangle to show where we zoom in next panel
ax2.add_patch(patches.Rectangle((128.63, 8.81),0.13,0.107,fill=False,color='b'))

plt.show()

Above, the left-hand panel shows a thin redshift slice (0.105 < z < 0.125, black symbols) among objects with redshifts from the spectroscopic SDSS DR13 sample (red symbols). We can see by eye some large-scale filamentary structures and overdensities. The blue rectangle shows a selected region where we zoom in the right-hand panel. On the latter, we further select a smaller region, which we will use in the next cell below.

## ZOOM IN AGAIN OVER AN EVEN SMALLER REGION
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5.5))

# plot all points in red (all redshifts)
ax1.scatter(result['plug_ra'],result['plug_dec'],s=30,color='r',marker='o',alpha=0.25)

# overplot in black objects in narrow redshift slice
ax1.scatter(result['plug_ra'][rz],result['plug_dec'][rz],s=50,color='black')

# Extent of RA, Dec (in degrees) to plot
xmin = 128.63
xmax = 128.76
ymin = 8.81
ymax = 8.917

ax1.axis([xmin, xmax, ymin, ymax])
ax1.set_xlim(reversed(ax1.set_xlim())) # flip the x-axis
ax1.set_xlabel("RA (degrees)", fontsize=20)
ax1.set_ylabel("Dec (degrees)", fontsize=20)

## SHOW DECaLS IMAGE (screenshot pre-made but could instead implement image cutout)
im = plt.imread('DECaLS_screenshot_zoomIn_labels.jpg')
ax2.imshow(im)
ax2.axis('off')

plt.show()

The left-hand panel shows the small region enclosed in the blue rectangle that we chose above (right-hand panel). The galaxies in black are in the same narrow redshift slice as defined previously (0.105 < z < 0.125). The right-hand panel is an image cutout of the same region of the sky from the LS sky viewer. The galaxies encircled correspond to the points in black, and some or perhaps most of them likely belong to a galaxy cluster.

Large-Scale Structures with LS Morphologies

There are many possible extensions to this work. For instance, one could plot again with symbols coded with object type (from LS) and/or class (from SDSS) and/or velocity dispersion (from SDSS) and/or other quantities. Here, we will start with the object "TYPE" from LS, related to the morphological shapes.

The object shape (2D light profile) is modeled by the Tractor (Lang, Hogg & Mykytyn) as part of the procedure to compute model photometry.

Possible shapes for LS DR3:

• PSF (point spread function: size will vary with the seeing of the observations)
• SIMP (“simple” galaxies: round, exponential profile with 0.45″ effective radius)
• EXP (exponential profile; spiral galaxies)
• DEV (deVaucouleurs profile; elliptical galaxies)
• COMP (composite deVaucouleurs+exponential at same centroid)

Please note that starting in DR4 and in subsequent data releases, the SIMP model as been replaced with a REX model (Round Exponential).

**Figure:** Images of galaxies including a nearby elliptical galaxy, a nearby spiral galaxy, and a QSO.

# Select redshift slice
rz = np.where((result['z'] >0.105) & (result['z']<0.125))

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6.5))

# plot all points in red (all redshifts)
ax1.scatter(result['plug_ra'],result['plug_dec'],s=3,color='r',marker='o',alpha=0.25)

# overplot in black objects in narrow redshift slice
ax1.scatter(result['plug_ra'][rz],result['plug_dec'][rz],s=10,color='black')

# Extent of RA, Dec (in degrees) to plot
xmin = 126.
xmax = 131.
ymin = 7.
ymax = 12.

ax1.axis([xmin, xmax, ymin, ymax])
ax1.set_xlim(reversed(ax1.set_xlim())) # flip the x-axis
ax1.set_xlabel("RA (degrees)", fontsize=20)
ax1.set_ylabel("Dec (degrees)", fontsize=20)


# Select redshift slice and per type
rdev  = np.where((result['type']=='DEV') & (result['z'] >0.105) & (result['z']<0.125))
rexp  = np.where((result['type']=='EXP') & (result['z'] >0.105) & (result['z']<0.125))
rcomp = np.where((result['type']=='COMP') & (result['z'] >0.105) & (result['z']<0.125))

# plot all points in red (all redshifts)
ax2.scatter(result['plug_ra'],result['plug_dec'],s=3,color='gray',marker='o',alpha=0.25)

# overplot in blue objects in narrow redshift slice
ax2.scatter(result['plug_ra'][rexp],result['plug_dec'][rexp],s=15,color='b')         # blue = EXP
ax2.scatter(result['plug_ra'][rdev],result['plug_dec'][rdev],s=15,color='r')         # red  = DEV
ax2.scatter(result['plug_ra'][rcomp],result['plug_dec'][rcomp],s=15,color='orange')  # orange = COMP

plt.axis([xmin, xmax, ymin, ymax])
plt.xlim(reversed(plt.xlim())) # flip the x-axis
plt.xlabel("RA (degrees)", fontsize=20)
plt.ylabel("Dec (degrees)", fontsize=20)
plt.show()

EXERCISE: Large-Scale Structures with Galaxy Colors

There are pre-computed colors available. The columns are described here: http://datalab.noirlab.edu/query.php?name=ls_dr3.dr3_specobj_dr13

Another possibility would be to plot again the galaxies spatial coordinates, but color-coded according to their photometric colors. This is left as an exercise for the user, but feel free to get in touch with the Data Lab Team if you have questions.

Large scale sctructure in 3D

We can explore filaments and clusters of galaxies better if we can plot them in 3D. We turn to a different area of the sky, and will query for a sample of SDSS galaxies in the near to slightly distant universe, and plot in 3D the cone containing them. To avoid conamination by the galactic plane of the Milky Way, we point our search cone at high galactic latitudes.

Data query and preparation

We select (mostly) SDSS galaxies within a 10-degree radius around a high galactic latitude direction, (ra,dec) = (160,45) degrees. We limit our search to positive redshifts between 0.02 and 0.3.

# Create the query to fetch the SDSS data from DataLab:

query = """
SELECT ra,dec,z
FROM sdss_dr17.specobj
WHERE q3c_radial_query(ra,dec,%s,%s,%s)
AND z between 0.02 AND 0.3
""" % (160,45,10)

print (query)

SELECT ra,dec,z
FROM sdss_dr17.specobj
WHERE q3c_radial_query(ra,dec,160,45,10)
AND z between 0.02 AND 0.3

Run the query to fetch the SDSS data from the ls_dr17.specobj table

%%time
selection = qc.query(adql=query, fmt='csv')

CPU times: user 33 ms, sys: 13 ms, total: 45.9 ms
Wall time: 1.29 s

Reformat output into a table

data = Table.read(selection, format='csv')  #dictionary
print("Number of galaxies in the sample: %d" % len(data))
data[:5]

Number of galaxies in the sample: 36331

Table length=5

ra	dec	z
float64	float64	float64
159.25732	54.941463	0.072250396
159.17507999999998	54.942007	0.14156331
162.24439	54.889704	0.14798966
162.50349	54.858997	0.17074445
162.42234	54.829701	0.18014126

Compute luminosity distance for every galaxy, using Planck2018 cosmology values

dist = cosmo.luminosity_distance(data['z'])
dist

$[337.39763,~691.83881,~726.1405,~\dots,~786.81545,~1396.6174,~1409.3111] \; \mathrm{Mpc}$

To plot the positions of each galaxy in true 3D space, we will convert the angular coordinates ra & dec, and the distance coordinate, to Cartesian coordinates X,Y,Z (all measured in Mpc from the coordinate system origin).

def get_cartesian(ra,dec,dist):
    # convert ra and dec to radians, since numpy expects this as arguments to trigonometric functions
    rarad = np.radians(ra)
    decrad = np.radians(dec)
    X = dist * np.sin(decrad) * np.cos(rarad)
    Y = dist * np.sin(decrad) * np.sin(rarad)
    Z = dist * np.cos(decrad)
    return X,Y,Z

X,Y,Z = get_cartesian(data['ra'],data['dec'],dist)

Plotting in 3D

To create an interactive 3D plot (one where we can zoom, pan, and rotate the scene), we will use the plotly package. The galaxies in our sample will be plotted using a 3D scatter routine. We also set some overall properties of the plot, such as the size of the markers, and a color map (we color each galaxy redder the further away it is).

trace = go.Scatter3d(
    x = X,
    y = Y,
    z = Z,
    mode = 'markers',
    marker = {
        'size'      : 0.7,
        'opacity'   : 0.5,
        'color'     : dist, 
        'colorscale': 'OrRd'
    }
)

data = [trace]

Next, we define camera location and the layout of the plot

# set up the view point
camera = dict(
    up = dict(x = 0, y = 0, z = 1),
    center = dict(x = 0.2, y = 0, z = 0),
    eye = dict(x = 0.6, y = -0.6, z = -1.0)
)

# set up the plot scene
layout = go.Layout(
    scene = dict(
        xaxis = dict(title = 'X',
                     backgroundcolor = 'black',
                     gridcolor = "rgb(40,40,40)"),
        yaxis = dict(title = 'Y',
                     backgroundcolor = 'black',
                     gridcolor = "rgb(40,40,40)"),
        zaxis = dict(title = 'Z',
                     backgroundcolor = 'black',
                     gridcolor = "rgb(40,40,40)"),
    ),
        scene_camera = camera,
        plot_bgcolor = 'black',
        paper_bgcolor = 'black',
        title = None,
        showlegend = False,
        width = 800,
        height = 800,
        autosize = False,
        margin = {'l':0, 'r':0, 'b':0, 't':0},
        dragmode = 'orbit'
)

# Draw the plot
plot_figure = go.Figure(data=data, layout=layout)
plot_figure.update_layout()

Explore the scene by:

• Zooming in and out (mouse scroll wheel up/down, or click mouse middle button and move mouse up/down, or select "zoom" mode via button in upper-right corner & left-click + drag mouse)
• Panning left/right and up/down (click mouse left button and drag)
• Change pan/rotate mode by selecting one of the buttons in the upper-right corner of the plot (pan, orbital rotation, turntable rotation)

Observe how the galaxies form compact local clusters, and on larger scales form huge filaments. This is the large scale structure of the universe.